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Abstract 

A  Join  Index  is  a  data  structure  used  for  processing  join  queries  in  databases.  Join  indices  use 
pre-computation  techniques  to  speed  up  online  query  processing  and  are  useful  for  data-sets  which 
are  updated  infrequently.  The  cost  of  join  computation  using  a  join-index  with  limited  buffer  space 
depends  primarily  on  the  page-access  sequence  used  to  fetch  the  pages  of  the  base  relations.  Given 
the  join-index,  we  introduce  a  suite  of  methods  based  on  clustering  to  compute  the  joins.  We  derive 
upper-bounds  on  the  lengths  of  the  page-access  sequences.  Experimental  results  with  Sequoia  2000 
data  sets  show  that  the  clustering  method  outperforms  the  existing  methods  based  on  sorting  and 
on  line-clustering  h  euristics . 
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1  Introduction 


The  join  operation  is  a  fundamental  operation  in  relational  databases,  and  substantial  work  has  been 
done  in  optimizing  join  operations  [11,  27].  A  join  index  [30.  34]  is  a  special  data  structure  that 
facilitates  rapid  join-query  processing.  For  data  sets  which  are  updated  infrequently,  the  join  index  can 
be  particularly  useful. 

The  join-index  is  typically  represented  as  a  bi-partite  graph  between  the  pages  of  encumbent  re¬ 
lations  or  their  surrogates.  When  the  number  of  buffer  pages  is  fixed,  the  join-computation  problem 
is  transformed  into  determining  a  page-access  sequence  such  that  the  join  can  be  computed  with  the 
minimum  number  of  redundant  page  accesses.  This  problem  has  been  shown  to  be  NP-hard  [26,  29], 
and  consequently,  it  is  unlikely  that  a  polynomial  time  solution  exists  for  this  problem.  Solutions  in  the 
literature  use  a  clustering  method  that  groups  pages  in  one  or  both  tables  involved  in  the  join  to  reduce 
total  page  accesses.  Available  heuristics  either  group  the  pages  of  a  single  table  via  sorting  [34]  or  use 
incremental  clustering  methods  [5,  7,  28]. 

Our  Contribution:  We  introduce  two  new  heuristics  for  this  problem.  One  heuristic  uses  the 
clustering  method  to  group  the  pages  in  one  table,  generalizing  the  sorting-based  heuristic  for  joins. 
The  other  heuristic  uses  clustering  for  the  pages  of  both  tables.  The  former  generalizes  and  outperforms 
the  sorting  heuristic,  while  the  latter  generalizes  and  outperforms  the  incremental  clustering  methods 
for  joins.  We  provide  a  formal  characterization  of  an  upper  bound  on  the  number  of  redundant  I/Os 
performed  by  our  approaches.  Experiments  with  the  Sequoia  2000  [33]  data-set  show  that  both  heuristics 
outperform  other  methods  when  the  memory  size  is  relatively  small.  The  proposed  approaches  are  useful 
for  computing  joins,  given  join-indices  for  large  database,  where  the  size  of  memory  is  small  compared 
to  the  sizes  of  the  individual  datasets. 

1.1  Join  Index:  Basic  Concept 

Consider  a  database  with  two  relations,  Facility  and  Forest  Stand.  Facility  has  a  point  attribute  repre¬ 
senting  its  location,  and  Forest  Stand  has  rectangle  attributes  that  represent  its  extent  by  a  bounding 
box.  The  polygon  representing  its  extent  may  be  stored  separately.  A  point  has  the  x  and  y  coordinates 
on  the  map.  A  rectangle  is  represented  by  points  that  represent  the  bottom  left  and  top  right  corners. 

In  Figure  1(a),  points  al,a2,a3,61, 62  represent  facility  locations,  and  polygons 
.41,  .42,  Bl,  B2.  Cl.  C2  are  the  bounding  boxes  that  represent  the  limits  of  the  forest  stands. 
The  circle  around  each  location  shows  the  area  within  distance  D  from  a  facility.  The  rectangle  around 
each  forest  boundary  represents  the  Minimal  Orthogonal  Bounding  Rectangle (M OB R.)  for  each  forest 
stand.  Figure  1(b)  shows  two  relations,  R  and  S,  for  this  data  set.  Relation  R,  represents  facilities 
via  the  attributes  of  a  unique  identifier,  R.ID,  the  location  (x,y  coordinates),  and  other  noil-spatial 
attributes.  Relation  5  represents  the  forest  stands  via  a  unique  identifier,  S.ID ,  the  MOBR,  and 
non-spatial  attributes.  MOBR, (AT/.,  Yll ,  A/?//,  Yvr),  is  represented  via  the  coordinates  of  the  lower-left 
corner  point  (A 'll, Yll)  and  the  upper  right  corner  point (Xur-Yux).  Now,  consider  the  following 
query:  Q:  55 Find  all  forest  stands  which  are  within  a  distance  D  from  each  facility”.  This  query  will 
require  a  join  on  the  Facility  and  Forest  Stand  relations,  based  on  their  spatial  attributes. 

A  spatial  join  algorithm  [2,  3,  4,  12,  25]  may  be  used  to  find  the  pairs  (Facility,  Forest-stand)  which 
satisfy  query  Q.  Alternatively,  a  join-index  may  be  used  to  materialize  a  subset  of  the  result  to  speed 
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up  processing  for  the  future  occurrence  of  Q  if  there  are  few  updates  to  the  spatial  data.  Figure  1(b) 
shows  a  join  index  with  two  columns.  Each  tuple  in  the  join  index  represents  a  tuple  in  the  table 
JO  I N(R,S,distance(R.  Location,  -S.M  OB  R)  <  D).  In  general,  the  tuples  in  the  join  index  may  also 
contain  pointers  to  the  pages  of  R  and  5  where  the  relevant  tuples  of  R  and  5  reside.  We  omit  the 
pointer  information  to  simplify  the  diagrams  in  this  paper. 
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(a)  Spatial  Attribute  of  R  and  S 


(b)  R  and  S  Relation  Table  and  Join  Index 


Figure  1:  Constructing  a  Join  Index  from  two  relations 

A  join  index  describes  a  relationship  between  the  objects  of  two  relations.  Assume  that  each  tuple  of 
a  relation  has  a  surrogate  (a  system-defined  identifier  for  tuples,  pages,  etc.)  which  uniquely  identifies 
that  tuple.  A  join  index  is  a  sequence  of  pairs  of  surrogates,  where  each  pair  of  surrogates  identifies  the 
result-tuple  of  a  join.  The  tuples  participating  in  the  join  result  are  given  by  their  surrogates.  Formally, 
let  R,  and  S  be  two  relations.  Then  consider  the  join  of  R  and  S  on  attributes  A  of  R.  and  B  of  S.  Then 
the  join  index  is  an  abstraction  of  the  join  of  the  relations.  If  F  defines  the  join  predicate,  then  the 
join  index  is  given  by  the  set  JI  =  {(rj,3j)\F(rj.A,  Sj.B)  is  true  for  r,  €  R.  and  Sj  £  S},  where  r*  and 
Sj  are  surrogates  of  the  ith  tuple  in  R  and  the  j th  tuple  in  5,  respectively.  For  example,  consider  the 
Facility  and  Forest  Stand  relational  tables  shown  in  Figure  1.  The  Facility  relation  is  joined  with  the 
Forest  Stand  relation  on  the  spatial  attributes  of  each  relation.  The  join-index  for  this  join  contains  the 
tuple  IDs  which  match  the  spatial  join  predicate. 

A  join  index  can  be  described  by  a  bipartite  graph  G  =  (Vj ,  Vj.  E),  where  V)  contains  the  tuple 
IDs  of  relation  R,  and  V-2  contains  the  tuple  IDs  of  relation  S.  Edge  set  E  contains  an  edge  (vr.vs)  for 
vr  £  R,  and  v9  £  S,  if  there  is  a  tuple  corresponding  to  {vr.v9)  in  the  join  index.  The  bipartite  graph 
models  all  of  the  related  tuples  as  connected  vertices  in  the  graph.  In  a  graph,  the  edges  connected  to 
a  node  are  called  the  incident  edges  of  that  node,  and  the  number  of  edges  incident  on  a  node  is  called 
the  degree  of  that  node. 

1.2  Page- Connectivity  Graph,  Page- Access  Sequence 

When  the  join  relationship  between  two  relations  is  described  at  the  page  level,  we  get  a  page-connectivity 
graph.  A  Page- Connectivity  Graph  (PCG)  [26]  Be  —  (V'i ,  V-2 ,  E)  is  a  bipartite  graph  where  vertex  set 
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V'i  represents  the  pages  from  the  first  relation,  and  vertex  set  V2  represents  the  pages  from  the  second 
relation.  The  set  of  edges  is  constructed  as  follows:  an  edge  is  added  between  page  (node)  v\  in  V\  and 
page  (node)  v.2  in  V2,  iff  there  is  at  least  one  pair  of  objects  (r?- .  Sj )  in  the  join  index  such  that  r<  €  v\ 
and  Sj  €  v2 .  Figure  2  shows  a  page-connectivity  graph  for  the  join  index  from  Figure  1(b).  Nodes  (a,  b) 
represent  the  pages  of  relation  R,  and  nodes  (A.B.C)  represent  the  pages  of  relation  5.  A  min- cut 
node  partition  [13,  22]  of  graph  Be  =  (Vj ,  V2:  E)  partitions  the  nodes  in  V  into  disjoint  subsets  while 
minimizing  the  number  of  edges  whose  incident  nodes  are  in  two  different  partitions.  The  cut-set  of 
a  min-cut  partition  is  the  set  of  edges  whose  incident  nodes  are  in  two  different  partitions.  Fast  and 
efficient  heuristic  algorithms  [20.  17]  for  this  problem  have  become  available  in  recent  years.  They  can 
be  used  to  cluster  pages  in  PCG. 


Figure  2:  Construction  of  a  Page-Connectivity  Graph  (PCG)  fr  om  a  Join  Index. 

A  join  index  helps  speed  up  the  join  processing,  as  it  keeps  track  of  all  of  the  pairs  of  tuples  which 
satisfy  the  join  predicate.  Given  a  join  index  JI.  one  can  use  the  derived  PCG  to  schedule  an  efficient 
page-access  sequence  to  fetch  the  data  pages.  The  CPU  cost  is  fixed,  as  there  is  a  fixed  cost  associated 
with  joining  each  pair  of  tuples,  and  the  number  of  tuples  to  be  joined  is  fixed.  I/O  cost,  on  the  other 
hand,  depends  on  the  sequence  of  pages  accessed.  When  there  is  limited  buffer  space  in  the  memory, 
some  of  the  pages  may  have  to  be  read  multiple  times  from  the  disk.  The  page-access  sequence( and  in 
turn  the  join-index  clustering  and  the  clustering  of  the  base  relation)  determines  the  I/O  cost. 

Example:  We  illustrate  the  dependency  between  the  I/O  cost  of  a  join  and  the  order  in  which 
the  data  pages  are  accessed  with  the  help  of  an  example,  using  the  page-connectivity  graph  shown  in 
Figure  2.  Assume  that  the  buffer  space  is  limited  to  allow  at  most  two  pages  of  the  relations  in  memory, 
after  caching  the  whole  page-connectivity  graph  in  memory.  Consider  the  two-page  access  sequences: 
(i)  (a,  A,  b,  B,  a,  C,  b)  and  (ii)  (a,  A,  b,  C,  a,  B).  Each  sequence  allows  the  computation  of  join  results 
using  a  limited  buffer  of  two  pages.  However,  in  the  first  case,  there  are  a  total  of  seven  page  accesses, 
and  in  the  second  case  there  are  a  total  of  six  page  accesses.  Note  that  the  lower  bound  on  the  number 
of  page  accesses  is  five,  as  there  are  five  distinct  pages  in  the  PCG.  However,  with  two  buffer  spaces, 
there  is  no  page-access  sequence  which  will  result  in  five  page  accesses.  This  is  because  the  cycle  (a,  A, 
b,  C,  a)  requires  that  at  least  three  pages  be  in  memory  to  avoid  redundant  page  accesses.  With  three 
buffer  spaces,  (a,  B,  A,  C,  b)  is  a  page-access  sequence  which  results  in  five  page  accesses. 
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1.3  Problem  Definition,  Scope,  Outline 

Given  that  the  I/O  cost  depends  on  the  page-aeeess  sequence,  the  following  optimization  problem  char¬ 
acterizes  the  problem  of  designing  efficient  algorithms  for  processing  joins,  given  a  join  index  and  a  fixed 
buffer  size.  This  problem  is  called  the  Optimal  Page-Access  Sequence  with  a  Fixed  Buffer  (OPAS-FB) 
problem  [26],  which  is  formally  defined  as  follows: 

OPAS-FB  Problem 

Given:  A  page-connectivity  graph  PCG  =  (V,  E).  representing  the  join  index,  and  a  buffer  of  size 
B  <  |V|. 

Find:  A  page-access  sequence. 

Objective:  To  minimize  the  number  of  page  accesses. 

Constraint:  Such  that  the  number  of  pages  in  the  buffer  is  never  more  than  B. 

For  example,  the  optimal  page-access  sequence  for  the  PCG  in  Figure  2  for  B  =  2  is  (a,  A,  b,  C,  a, 
B),  which  results  in  six  page  accesses. 

The  OPAS-FB  problem  is  known  to  be  NP-hard  [26,  29],  and  heuristic  solutions  have  been  proposed 
in  the  literature  for  solving  this  problem.  The  heuristics  in  the  literature  can  be  broadly  divided  into 
two  groups,  namely  asymmetric  single-table  clustering  and  symmetric  two-table  on-line  clustering.  We 
describe  the  relevant  literature  and  our  contribution  in  sections  2  and  3. 

Scope:  In  this  paper,  we  focus  on  the  OPAS-FB  problem.  We  do  not  address  the  update  problems 
associated  with  managing  join  indices.  Also,  the  base-relation  clustering  and  tuple-level  join-index 
optimization  are  out  of  the  scope  of  this  paper. 

Outline:  The  rest  of  the  paper  is  organized  as  follows.  Section  2  discusses  asymmetric  single-table 
clustering  methods  and  proposes  our  first  approach,  Asymmetric  Clustering(AC),  evaluating  its  perfor¬ 
mance  with  the  Sorting  heuristic.  Section  3  describes  two-table  on-line  clustering  from  the  literature. 
In  Section  4,  we  propose  Symmetric  Clustering  (SC).  In  Section  5,  we  compare  the  proposed  methods, 
AC  and  SC,  with  traditional  algorithms  for  the  OPAS-FB  problem.  Section  6  includes  a  summary  and 
future  research  directions. 

2  Asymmetric  Methods 

The  main  approach  within  asymmetric  single-table  clustering  is  based  on  sorting  the  join-index  on 
one  of  the  join  keys.  In  the  following  discussion,  let  R  and  S  be  the  two  relations,  with  ,11  being  the 
join  index.  The  sorting-based  asymmetric  heuristic  presented  in  [34]  reads  as  much  as  possible  of 
the  join  index  (JI)  and  one  relevant  relation  (R  semi-join  JI)  into  memory.  Here  JI  is  assumed  to  be 
sorted  on  relation  R..ID.  To  reduce  redundant  accesses  to  S,  access  to  S  is  clustered  by  sorting  the  list 
of  all  of  the  surrogates  from  S  that  are  related  to  the  subset  of  the  join-index  in  memory.  This  heuristic 
ensures  that  no  redundant  accesses  are  performed  on  relation  R,  but  it  may  incur  redundant  accesses 
to  the  second  relation.  The  Sorting-based  heuristic  is  most  suited  to  the  applications  that  have  totally 
ordered  join-keys.  Rigorously  speaking,  the  sorting-based  heuristic  sorts  the  surrogates(e.g.  system- 
defined  identifiers  for  pages)  rather  than  the  join-key  attributes.  If  tables  are  sorted  by  the  respective 
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join-keys,  then  surrogates  for  the  pages  in  a  table  may  be  ordered  by  the  lowest  key-value  for  any  tuple 
in  the  page.  This  reduces  redundant  page  I/O  in  computing  joins  using  a  join-index,  for  join-keys  with 
totally  ordered  domains.  Since  multi-dimensional  domains,  e.g.  spatial  data  types,  do  not  have  natural 
total-order,  sorting  surrogates  may  not  be  as  effective  for  computing  spatial-joins  using  join-indices.  We 
propose  Asymmetric  Clustering  to  address  this  problem.  Asymmetric  clustering  uses  the  entries  in  the 
join-index  for  grouping  pages  of  one  relation,  say  R,  based  on  their  interaction  with  the  pages  in  the 
other  relation,  say  S.  If  the  join-index  (e.g.  Figure  1(b))  represents  the  summary  of  a  spatial  join,  then 
the  pages  of  R  are  clustered  using  their  spatial  relationship  with  the  pages  of  S,  and  the  proposed  method 
is  called  Asymmetric  Spatial  Clustering. 


2.1  Basic  Idea  Behind  Asymmetric  Clustering(AC) 

The  example  in  Figure  3  highlights  the  different  approaches  of  AC  and  the  sorting  heuristic.  Figure  3(a) 
shows  a  49-node  PCG,  with  numbers  1-24  and  letters  A-Y  corresponding  to  the  surrogates  of  the  two 
page-level  relations.  The  figure  can  represent  a  spatial-join  computation  between  two  layers  of  geographic 
data  consisting  of  small  polygons.  The  pages  from  each  relation  may  overlap  pages  from  the  other. 
Consider  a  memory  with  seven  pages  for  buffering  the  pages  of  R  and  5  relations.  There  may  be 
additional  buffering  pages  for  managing  the  result,  index,  etc.  The  AC  heuristic  used  a  different  clustering 
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Figure  3:  Example  of  the  AC  and  Sorting  heuristic 

of  pages  of  the  first  relation,  relative  to  the  sorting- based  approach.  Figure  3(b)  and  3(c)  show  the 
clusters  of  pages  of  the  first  relation  that  are  used  by  AC  and  the  sorting-based  method.  The  pages  of 
R,  are  numbered  1-24,  and  within  a  cluster  are  annotated  by  a  common  symbol.  For  example,  nodes 
1,2, 3.4, 5.6  in  Figure  3(b)  are  all  circled,  denoting  that  these  are  loaded  together  by  sorting.  Similarly, 
nodes  7,8,9. 10, 1 1 , 12  are  annotated  with  a  hexagon,  and  so  on.  Visually,  one  can  verify  that  the  clusters 
used  by  AC  are  spatially  cohesive.  The  number  of  pages  of  the  second  relation  which  have  edges  to 
multiple  clusters  of  pages  in  the  first  relation  yield  redundant  I/O.  One  may  consider  using  other  space¬ 
filling  curves(e.g.  Z-order.  Hilbert)  to  improve  the  performance  of  the  sorting-based  method.  However, 
min-cut  graph  partitioning,  the  method  used  by  AC,  outperforms  space-filling  curves  in  the  clustering 
of  non-uniformly  distributed  spatial  data,  as  shown  in  our  previous  work  [31,  32]. 

The  sorting  heuristic  clusters  the  nodes  of  the  first  page-level  relation  and  then  loads  the  pages 
in  the  sorted  order.  The  loading  sequence  for  the  example  in  Figure  3  is  shown  in  Table  2.  With  a 
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buffer  size  of  seven,  the  sorting  heuristic  will  load  pages  {1,2, 3,4,5, 6}  from  the  first  relation  in  the 
first  six  buffers  and  then  will  load  one  page  at  a  time  from  the  set  of  pages,  {-4,  B.C,  D.  E,  F,  G,  II.  /}, 
in  the  7th  buffer.  For  the  next  round,  it  loads  {7,8,9,10,11,12}  in  the  first  six  buffers,  and  then 
{E,  F,  G ,  II,  I,  J,  K,  L.  M ,  N,  O,  P}  one  at  a  time  in  the  7th  buffer,  and  so  on,  as  shown  in  Table  2.  The 
sorting  heuristic  results  in  17  redundant  I/Os,  with  a  total  of  66  I/Os. 
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First  Six.  Buffers 

The  7th  buffer 

First  Six  Buffers 

The  7th  buffer 

1 

1,2, 3.4.5, 6 

A,B,C,D,E,F,G,H,I 

1,2,4,5,9,10 

A.B.C.D.F.G.H.L.M.N 

2 

7,8,9, 10,1 1 ,12 

E,F,G,H,I,J,K,L,M,N,0,P 

3,7,8,13,14,19 

B.E.F.J.K.L.Q.R.V 

3 

13,14,15,16,17,18 

J,K,L,M,N,0,P,Q,R,S,T,lJ 

6,11 ,12,17,18,22 

D,H  .1  ,N  .O.P.T.U  .X 

4 

19,20,21 ,22,23,24 

Q,R,S,T,U,V,W,X,Y 

15,1 6,20,21 ,23,24 

L.M.N.R.S.T.VAV.X.Y 

Total  I/O 

66 

62 

Table  2:  Loading  Sequence  of  Sorting  and  APG  for  Figure  4 


The  AC  clusters  the  nodes  of  the  first  page-level  relation  according  to  their  connections  with  the  sec¬ 
ond  relation.  The  clustering,  as  shown  in  Figure  3(c),  with  a  buffer  size  of  seven,  provides  four  clus¬ 
ters.  Note  that  these  clusters  are  different  from  the  page-clusters  used  in  sorting.  The  AC  heuristic 
will  load  pages  {1,2,4,5,9,10}  of  the  first  relation  in  the  first  six  buffers,  then,  one  by  one,  will  load 
{A,  B,  C,  D,  F,  G,  H,  L,  M,  Ar}  in  the  7th  buffer.  In  the  next  round,  it  loads  {3, 7, 8, 13, 14, 19}  together 
into  the  first  six  buffers,  and  then  {B.  E,  F.  J.  K.  L,  Q,  R.  V}  one  at  a  time  into  the  7th  buffer.  Table  2 
shows  the  total  loading  sequence.  The  AC  results  in  13  redundant  I/Os,  with  a  total  of  62  I/Os.  The 
difference  of  four  I/Os  out  of  66  in  this  example  may  not  look  large.  However,  the  relative  difference  in 
I/Os  using  the  sorting  and  clustering  methods  will  increase  with  an  increase  in  data-set  size.  This  linear 
characteristic  of  sorting  yields  poor  clustering  and  limits  the  savings  in  redundant  I/Os. 


2.2  Description  of  Asymmetric  Clustering  Method 

The  goal  of  asymmetric  clustering  methods  is  to  cluster  the  pages  of  one  relation,  given  the  join-index  or 
its  PCG.  This  can  be  formalized  as  amin-cut  hypergraph-partitioning  problem.  The  pages  of  a  relation 
will  form  the  nodes  of  the  hypergraph.  Each  page  p  of  the  other  relation  will  form  a  hyperedge,  covering 
all  pages  of  the  first  relation  connected  to  p  in  PCG.  Partitioning  the  nodes  in  this  hypergraph  will  form 
a  group  of  pages  of  the  first  relation  that  can  be  loaded  together.  Minimizing  cut  hyperedges  during 
partitioning  reduces  the  number  of  pages  of  the  second  relation  that  will  need  to  be  loaded  into  memory 
multiple  times. 

Consider  the  example  spatial-join  problem  depicted  in  Figure  4(a)  with  two  point  data-sets,  (a,b,c,d) 
and  (A.B.C.D).  Assume  a  blocking  factor  of  1  to  simplify  the  example.  The  PCG  of  the  join-index 
for  Distance^,  j)  <  is  shown  in  Figure  4(c)  using  the  overlay  and  distance  buffer  information.  The 
nodes  of  the  hypergraph  shown  in  Figure  4(d)  consist  of  the  nodes  of  relation  E,  i.e.  (a,b,c,d).  The 
hvperedges  represent  nodes  (A,B,C,D)  of  S.  The  hyperedge  corresponding  to  A  connects  a  and  c,  since 
(A, a)  and  (A,e)  satisfy  the  join  predicate.  The  partition  ((a,c),(b,d))  has  no  cut  hvperedges,  and 
computing  the  join  using  it  will  result  in  no  redundant  I/O  with  loading  sequence  (a,c,A,C,b,d,B,D), 
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if  3  buffers  are  available  to  hold  the  pages  of  two  relations.  In  contrast,  the  partition  ((a,b),(c,d)) 
cuts  all  four  hvperedges,  and  computing  this  join  will  yield  four  redundant  I/Os  with  loading  sequence 
(a,b,A,C,B,D,c,d,A,C,B,D),  if  only  3  buffers  are  available  to  hold  the  pages  of  two  relations. 


(a)  Dataset  R  (b)  DatasetS  (c)  Overlay  and  Join  (d)  Graph  PCG  (d)  Hypergraph  model 

distance  <  L/  /2  (one  object)/page 


Figure  4:  Construction  of  a  one-sided  hypergraph  from  the  data  set 
We  formally  describe  AC  now,  via  the  following  pseudo-code. 

AC  Algorithm 

Input:  G  =  (Vr,Va,E)  is  a  page  connectivity  graph 

Output:  S  =<  P\  ,/J2, Pr  >  is  a  page  access  sequence  with  r  >  |Vr|  +  |Vs|(P*s  need  not  be  distinct) 
assert (|Vj*  |  <  |V*|); 

assert [B  >  2);  /*  number  of  buffers  */ 

HGr(Vr,  HEr)  =  Derive  Hypergraph(<3);  /*  HGr  is  a  hypergraph.  j  UEr |  =  [V4,  [  */ 

/*  For  each  node  in  |Va|,  build  a  hyperedge  to  encompass  all  of  its  corresponding  nodes  in  Vr  */ 

PSe.tr  =  h  M  et  is-  Part  it  ion  ( //Gr ,  B  -  1)  /*  PSetr  is  the  set  of  partitions  */ 
i=0; 

while  ((/J;r  =SelectUnprossed Partition (/J.S'eq))!=NULL)  /*  Select  the  un-processed  partition  */ 

{ 

AddPageSequencef.S’.  Plr  )]/*  Add  all  the  nodes  in  P;r  into  the  loading  sequence  */ 

Pi,  =  Sort-Eliminate-Dup(G.  P;,,); 

f*  Sort  and  eliminate  the  duplicated  nodes  in  Vs  of  G  which  connect  to  nodes  in  P;,,  */ 

AddPageSequence(5,  Pi,)',/*  Add  all  the  nodes  in  Pi,  into  the  loading  sequence  */ 

Pi,,  .flag  =  "processed";  /*  Mark  this  partition  as  ” processed */ 

i++; 

} 

The  first  step  of  the  AC  algorithm,  i.e.  DeriveHp  ergraph(G),  creates  a  hvpergraph  from  a  given  page 
connectivity  graph  G.  Nodes  of  the  first  relation  R  form  the  nodes  of  the  hypergraph.  For  each  node  v 
of  the  second  relation,  it  builds  a  hvperedge  to  encompass  a  set  of  nodes  on  the  first  relation (/?)  that  are 
connected  to  v  in  G.  Next,  AC  partitions  this  hyper-graph  using  the  min-cut  hvper-graph  partitioning 
algorithm,  hMetis  [17,  18,  19],  which  is  a  multi-level  hypergraph-partitioning  algorithm  that  has  been 
shown  to  produce  high  quality  bi-sections  on  a  wide  range  of  problems  that  arise  in  scientific  and  VLSI 
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applications.  hMetis  minimizes  the  (weighted)  hyper-cut  and  thus  tends  to  create  partitions  in  which 
connectivity  among  the  vertices  in  each  partition  is  high,  resulting  in  good  clusters.  Finally,  AC  loads 
each  partition  in  the  primary  relation  and  its  connected  nodes  in  the  second  relation,  one  by  one,  to 
compute  the  join.  The  I/O  cost  of  AC  can  be  characterized  via  the  following  lemma: 

Lemma  1  Given  a  partition  { ly, ,  Vr2. . . . ,  VTp  }  ofVr,  i.e.  pages  of  relation  R,  from  the  page- connectivity 
graph  PCG  =  (Vr,V9,E),  there  is  a  page-access  sequence  of  length  K  =  |  Vr  |  +  E«ev  f(v)  Pr°eess  the 
spatial  join,  where  f(v)  denotes  the  number  of  partitions  ofVr  that  have  an  edge  to  node  v  in  Ve. 

Proof:  A  node  v  in  V»  is  connected  to  f(v)  partitions  of  Vr.  Therefore,  the  node  v  in  Ve  has  to  be 
loaded  f(v)  times  into  the  buffer  to  compute  the  spatial  join.  The  total  number  of  redundant  I/Os  is 
Evev.  (/W  ~  !)•  The  total  I/O  cost  =  |V'r|  +  |V,|  +  Ewgv.  (/(«-’)  "  T)  =  \VA  +  \V8\  +  E,,ev.  f(v)  ~  \v«\  = 
\vr\  +  zv€Viif(v)m 

We  note  that  the  min-cut  hypergraph-partitioning  algorithm,  e.g.  hMetis,  minimizes  the  number  of 
hyperedges  connecting  nodes  across  clusters.  This  does  not  directly  minimize  E„ev.  f(v)>  as  ^  does 
not  distinguish  between  a  hyperedge  spanning  four  clusters  and  another  spanning  two  clusters.  While 
AC  outperforms  the  sorting- based  heuristic  already,  the  performance  of  AC  will  improve  when  better 
algorithms  for  hypergraph  partitioning  are  available  which  minimize  the  total  number  of  cuts  on  cut- 
hyperedges.  We  plan  to  explore  this  in  future  work. 

2.3  Comparison  of  Asymmetric  Approaches:  Sorting  vs.  AC 
2.3.1  Experiment  Design 

We  now  compare  the  performance  of  the  Sorting  heuristic  and  AC,  using  a  join  index  derived  from 
the  Sequoia  2000  [33]  dataset.  The  Point  table  contains  62,584  California  place  names  with  their 
associated  loeations(Longitude  and  Latitude),  extracted  from  the  US  Geological  Survey’s  Geographic 
Names  Information  System (GNIS).  The  Polygon  table  contains  4,388  records,  representing  Cropland 
and  Pasture  land  use  in  California.  Throughout  Section  2.3  and  5,  the  Point  and  Polygon  tables  will  be 
referenced  as  R  and  5,  respectively.  We  plot  the  point  and  polygon  data  set  of  California  records  as  in 
Figure  5. 

Now,  consider  the  following  queries: 

Q.A.  ’’For  each  place  in  the  Point,  table,  find  N  nearest  croplands  from  the  Polygon  table”. 

Q.B.  ’’For  each  place  in  the  Point  table,  find  all  croplands  which  are  within  a  distance  D”. 


The  spatial  join  of  these  two  queries  produces  sets  of  join  indices  and  such  join  indices  are  of  interest 
in  spatial  data  mining  for  neighborhood  indexing  [6].  The  value  of  N  and  D  can  be  in  creased /decreased 
for  adjusting  the  edge  ratio  [5].  Give  a  join  graph  G  -  (Vn,  Vs.  E).  the  edge  ratio  of  G,  denoted  by  0, 
is  defined  as  the  ratio  of  the  total  number  of  edges  in  G  to  the  maximum  possible  number  of  edges  in  G 
if  it  is  a  fully  connected  graph;  i.e.,  0  =  The  edge  ratio  provides  a  measure  of  the  page-level 

!  v  ft  j  j  1  5  j 

join  selectivity. 

The  variable  parameters  are  the  Buffer  size,  Page  size  and  Edge  ratio.  The  metric  of  evaluation  is  the 
number  of  page  accesses  required  by  each  algorithm  to  implement  the  join.  The  edge  ratio  is  controlled 
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0 


(a)  Polygons  for  Landuse 


(b)  GNIS  Points 


Figure  5:  Example  of  the  Sequoia  2000  data  set 


by  N  and  D.  The  number  of  nearest  neighbors,  N,  is  varied  from  1  to  5,  yielding  an  edge-ratio  of  0.002  to 
0.005.  The  side-size  of  range  queries,  D,  is  varied  from  400  to  4800  units  where  the  extent  of  California 
is  almost  1.2*  10°  units  (North-South)  x  0.8  *  106  units  (East- West).  This  yields  an  edge-ratio  of  0.002 
to  0.003. 

Page  size  represents  the  size  of  disk  blocks  and  memory  pages.  Different  values  of  page  sizes  include 
2,  4,  8,  16,  32,  64  Kbytes.  The  size  of  the  records  in  the  point  table  is  64  bytes.  The  blocking  factor 
for  the  Point  table  is  the  ratio  of  page  size  and  record  size.  Point  records  are  spatially  clustered  in  the 
pages  of  the  point  table.  The  records  in  the  Polygon  table  are  of  variable  size.  The  size  of  a  record  in 
the  Polygon  table  is  16+  32*  (#  of  Points  in  the  Polygon)  bytes.  The  number  of  points  in  a  polygon  can 
vary  from  a  dozen  to  a  few  thousands,  and  a  large  polygon  may  span  multiple  pages. 

The  buffer  size  represents  the  ratio  of  available  memory  size  as  a  fraction  of  the  size  of  the  Point 
table,  which  is  the  smaller  of  the  two  tables.  Memory  buffer  size  varies  from  4%  to  20%  of  the  size  of 
the  smaller  table. 

Figure  6  shows  the  various  process  steps  of  the  experiment  design.  From  the  base  point  and  polygon 
tables,  we  derived  families  of  join  indices  for  queries  Q.A.  and  Q.B.  for  different  values  of  edge  ratio. 
Next,  a  set  of  page-connectivity  graphs(PCGs)  are  generated  for  each  join  index,  given  different  values 
of  page  size.  The  page-connectivity  graphs  are  input  to  the  ”  Page- Access-Sequence  Generator,”  which 
simulates  the  behavior  of  OPAS-FB  algorithms  (i.e.  Sorting  and  AC)  for  a  given  buffer  size.  The  page- 
access  sequence  and  total  page  I/Os  are  tracked  for  each  combination  of  join  algorithm,  page  size,  buffer 
size,  and  edge  ratio  to  derive  the  experiment  results. 

2.3.2  Experiment  results 

Figures  7  and  8  show  the  comparison  between  the  AC  and  Sorting  heuristic  for  range-query  join-indices 
and  N-nearest -neighbor  join-indices,  respectively.  The  AC  method  is  uniformly  better  than  the  Sorting 
heuristic. 

Figures  7(a)  and  8(a)  show  the  impact  of  page  size,  which  we  van-  from  2  kbytes  to  64  kbytes.  The 
page  size  and  the  number  of  page  accesses  are  shown  in  logarithm  scale(base  two).  As  the  page  size 
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Figure  6:  Experimental  setup  and  design. 


increases,  the  number  of  pages  decreases,  and  clustering  efficiency  improves  for  all  methods,  reducing 
the  performance  gap  between  the  two  methods. 

Figures  7(b)  and  8(b)  show  the  effect  of  buffer  size  (as  a  fraction  of  the  size  of  the  smaller  relation) 
on  the  I/O  performance  of  AC  and  the  sorting-based  method.  As  long  as  the  buffer  is  smaller  than 
the  smaller  of  the  two  relations  involved  in  the  join,  both  AC  and  the  sorting- based  approach  use  most 
of  the  buffers  to  load  the  pages  of  only  one  relation.  The  difference  in  performance  comes  from  their 
difference  in  clustering  ability:  AC  has  a  lower  I/O  cost  than  Sorting. 

Figures  7(c)  and  8(c)  show  the  effect  of  the  edge  ratio.  AC  uniformly  outperforms  the  Sorting- based 
approach.  The  gap  between  the  performance  of  the  two  methods  does  not  show  any  trend. 


logfPage  size  (Kbytes)) 


(Buffer  size/smaller  (size  of  R,  size  of  S))*1 00 


Size  ofrange  query 


(a)  Page  size:  2K  -  64  K,  Buffer  size: 
10%  of  the  smaller  relation.  Range 
query  size  =  3800 


(b)  Buffer  size:  5%  -  20%  of  the 
smaller  relation.  Page  size:  2K, 
Range  query  size  =  3800 


(c)  Range  query  size  =  400  - 
4800(0=0.002-0. 003),  Buffer  size: 
10%  of  the  smaller  relation.  Page 
size:  2K 


Figure  7:  Range  Query  Join-index:  The  effect  of  page  size,  buffer  size,  and  number  of  nearest 
neighbors  on  the  AC  and  Sorting  heuristic 
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log(Page  size  (Kbytes)) 

(a)  Page  size:  2K  -  64K,  Buffer  size: 
5%  of  the  smaller  relation,  No  of 
nearest  neighbors=3 


4  6  8  10  12  14  16  18  20  22 
(Buffer  size/smaller  (size  of  R,  size  of  S))*1 00 

(b)  Buffer  size:  5%  -  20%  of  the 
smaller  relation,  Page  size:  2K,  No 
of  nearest  neighbors=3 


(c)  No  of  nearest  neighbors=l- 
5(0=0.002-0.005),  Buffer  size:  5% 
of  the  smaller  relation,  Page  size: 
2K 


Figure  8:  Nearest  Neighbor  Join-index:  The  effect  of  page  size,  buffer  size,  and  number  of  nearest 
neighbors  on  the  AC  and  Sorting  heuristic 

3  Symmetric  Methods 

While  AC  is  an  improvement  over  the  Sorting- based  method  for  spatial  joins,  it  has  a  few  drawbacks.  For 
example,  its  buffer  utilization  can  be  poor,  since  it  gives  almost  the  entire  buffer  space  to  one  relation.  We 
illustrate  this  with  the  help  of  the  spatial  join  problem  shown  in  Figure  9.  Figure  9(a)  shows  a  polygon  set 
with  6  polygons,  R0..R5,  and  a  point  data  set  with  G  points.  The  adjacency  matrix  Mpcc  representation 
of  a  join-index  is  shown  in  Figure  9(b),  along  with  the  page  access  sequence  for  the  sorting- based 
algorithm  with  three  memory  buffers.  Sorting  requires  1G  I/Os,  including  4  redundant  I/Os  on  SI,  S2, 
S3,  and  S4.  using  a  page-access  sequence  of  720, 721.  SO,  SI,  S2,  722. 723,  SI,  S2,  S3,  S4. 724, 725,  S3,  S4,  S5. 

A  symmetric  method  may  alternate  between  the  pages  of  the  two  relations,  as  shown  in  Fig¬ 
ure  9(c),  to  compute  the  join  with  12  I/O  (i.e.  no  redundant  I/O)  using  a  page-access  sequence  of 
720,  SO,  SI,  721,  S2, 722,  S3, 723,  S4, 724,  So. 725.  This  property  can  be  generalized  to  other  adjacency  ma¬ 
trices  with  only  B-diagonal  entries,  where  {Mrjcc[i-  j]  =  1}  =4-  {| i  —  j |  <  and  B  is  the  number 

of  buffers  available  for  pages  of  R  and  S.  The  indices  i  and  j  refer  to  the  row-indices  and  column-indices. 
The  symmetric  method  can  process  the  B-diagonal  entries  of  an  adjacency  matrix  with  no  redundant 
I/Os,  given  B  buffers  for  R.  and  S. 


The  main  approaches  in  symmetric  two-table  clustering  are  based  on  either  the  Traveling  Salesman 
Problem  heuristic  or  on  incremental  clustering.  A  traveling  salesperson (TSP)-based  heuristic  [10]  uses 
a  complete  graph  constructed  by  taking  the  nodes  of  one  relation  as  the  nodes  of  the  graph.  The  weight 
on  an  edge  between  nodes  a  and  b  denotes  the  number  of  page-accesses  required  to  fetch  all  of  the 
neighbors  of  b,  given  that  all  of  the  neighbors  of  a  are  in  memory.  This  method  requires  a  large  amount 
of  memory,  as  the  complete  graph  grows  quadratically  with  the  number  of  nodes  in  the  smaller  of  the 
relations.  Incremental  clustering  is  based  on  selecting  the  next  page  or  the  next  set  of  pages  to  be  fetched 
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Figure  9:  Comparison  of  symmetric  and  asymmetric  methods 


into  memory,  given  the  pages  in  the  buffer  and  the  remaining  edges  to  be  processed  in  the  bi-partite 
page-connectivity  graph.  The  selection  is  often  based  on  the  number  of  neighbors  in  the  memory  buffers 
and  the  number  of  neighbors  on  the  disk.  Details  of  actual  heuristics  follow. 

Symmetric  Heuristic:  FP  was  proposed  by  Fotouhi  and  Pramanik  [7].  The  buffer  is  initialized 
with  a  node  which  has  the  smallest  degree  in  the  page-connectivity  graph.  The  memory  buffer  is  added 
with  the  largest  resident-degree  node.  The  resident  degree  of  node  .4  is  the  number  of  nodes  which  are 
connected  to  .4  and  are  in  memory  buffers.  If  more  than  one  node  has  the  largest  resident  degree  ,  the 
algorithm  chooses  the  one  with  the  smallest  non-resident  degree.  The  non-resident  degree  of  a  node  A 
is  equal  to  totaLdegree(A)  —  resident  jkgree{ A).  When  the  buffer  is  full,  a  node  that  has  the  smallest 
number  of  edges  with  the  nodes  on  the  disk  can  be  swapped  out. 

Symmetric  Heuristic:  OM.  developed  by  Omiecinski  [28],  is  designed  specifically  for  bipartite 
join  graphs.  Initially,  load  a  pair  of  nodes  (ri,Sj),  where  r*  G  R  and  Sj  G  S,  from  the  page- connectivity 
graph  in  the  memory  buffers,  such  that  (a)(r,-,  Sj)  is  connected  and  (b)the  sum  of  the  degree  of  r*  and  Sj 
is  minimal.  In  each  iteration,  an  out-of-memory  least-non-residential-degree  neighbor  p  of  an  in-memory 
lowest-non-residential-degree  node  q  is  selected  to  be  swapped  in.  If  the  memory  buffers  are  full,  then 
the  lowest-non-residential  degree  node  r  which  is  not  connected  to  p  in  PCG  is  swapped  out.  Node  r 
may  come  back  to  memory  within  the  next  few  iterations. 

Symmetric  Heuristic:  Chan  [5]  generalized  the  OM  heuristic  by  swapping  in  a  set  of  nodes, 
i.e.  segments,  in  each  iteration.  The  set  of  nodes  selected  for  an  iteration  are  the  unprocessed  on-disk 
neighbors  of  the  lowest-non-residential  degree  node  n.  either  in  memory  or  on  disk.  Node  n  can  be 
processed  and  swapped  out  of  the  memory  buffers  at  the  end  of  the  current  iteration.  If  the  memory 
buffers  do  not  have  enough  empty  space,  then  the  page  with  the  lowest  number  of  a  non-residential 
degree  is  swapped  out. 

Most  symmetric  methods  proposed  in  the  literature  are  incremental,  considering  local  information 
in  PCG.  We  propose  a  Symmetric  Clustering(SC)  method,  which  exploits  global  information  across  the 
entire  PCG.  SC  is  described  in  section  4. 


3.1  An  example 


We  use  an  example  to  illustrate  the  differences  between  various  heuristics  for  computing  joins,  given  a 
join-index.  Readers  are  warned  that  this  example  is  a  bit  detailed  to  bring  out  the  differences  between 
various  methods.  On  the  first  reading,  one  may  gloss  over  the  details  of  Section  3.1,  and  simply  look  at 
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Table  3  to  get  the  summary.  We  have  tried  hard  to  find  an  example  which  could  both  explain  our  method, 
i.e.  SC.  and  explain  the  differences  between  all  of  the  methods.  We  have  not  been  able  to  find  anything 
simpler.  Figure  10(a)  shows  the  polygon-clusters  in  R  and  5  relations  with  their  overlays.  Some  polygon- 
clusters  have  one  polygon;  others  have  two  polygons,  for  example,  R.Q,  Rl, . . . ,  Ro,  51, 52, . . . ,  55,  and 
58,59,..  .,513.  Polygon  clusters  are  natural  in  geographic  data  as  well.  For  example,  the  boundary 
of  the  United  States  will  be  represented  by  a  collection  of  polygons  representing  the  Mainland,  Alaska, 
Hawaii,  etc. 
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Dark  region=edges  within  a 
partition.  Lightly  shaded  re- 
gion=edges  within  B-diagonal 


Figure  10:  Connectivity  graph  of  two  relations 

The  spatial-join  relationship  shown  uses  geometric  as  well  as  topological  representation.  Figure  10(a) 
shows  an  overlay  of  the  two  data-sets  to  provide  a  visual  representation  of  join  relationships.  Shaded 
areas  are  provided  for  realism.  One  may  imagine  a  beautiful  city  with  many  parks  and  lakes(shaded 
areas)  which  break  the  continuity  of  streets  (objects  in  R)  and  avenues  (objects  in  5).  Figure  10(b) 
provides  adjacency  matrix  representation  of  the  page-connectivity  graph  with  one  object  per  page.  To 
simplify  the  example,  we  assume  a  unit  blocking  factor,  i.e.  one  polygon- cluster  or  road  per  disk  page. 

A  summary  of  the  behavior  of  alternative  methods  is  shown  in  Figure  11,  which  has  five  different 
pieces.  The  adjacency  matrix  representation  of  the  join  index  is  reproduced  from  Figure  10(b)  to  facilitate 
understanding.  The  IDs  of  the  polygon  clusters  in  R  and  S  appear  immediately  to  the  left  of  the  matrix. 
The  two  vectors  left  of  and  below  the  node-identifiers  list  the  degree  of  each  node  in  PCG.  For  example, 
the  degree  of  R13  is  6.  The  nodes  with  the  highest  degree  (13)  are  R5  and  S8,  followed  by  R12,  which 
has  a  degree  of  12. 

The  remaining  two  pieces  of  Figure  11  present  the  summary  behavior  of  various  algorithms  in  terms 
of  page-access  sequence.  These  tables  show  the  rank-order  of  polygon-clusters  (i.e.  Pages  assuming  unit 
blocking  factor)  from  R  and  S  in  their  respective  page-access  sequences.  For  example,  the  page  access 
sequence  for  OM  is  R0,S0,S1,S2,S8,R1,S3,R2,S4,  R3,S5,R4,R5,R6,..  and  so  on.  Multiple  ranks  for  anode 
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Figure  11:  Comparison  of  different  methods 


indicate  redundant  I/O.  For  example,  OM  loads  R4,R.6,R.7,R,8,  and  S7  twice  in  memory  with  rankings  of 
(12,30), (14, 31), (15, 24), (17, 32),  and  (25,33).  This  information  is  further  summarized  in  Table  3,  which 
shows  total  I/O,  nodes  with  redundant  I/O,  and  min/max/average  degree  of  nodes  with  redundant  I/O. 
More  details  are  available  in  Appendix  A  and  B. 


Method 

AC 

FP 

OM 

Chan 

SC 

Total  I/O 

40 

45 

33 

31 

Nodes  with  redundant  I/O 

S2-S13 

R1-R4,  R6,  R7,  R9. 
RIO,  S0-S5,  S7,  S8 

R4,R6,R7, 

R8.S7 

S0-S6 

R12,  Ro, 
S13 

Degree  of 

nodes  with 
redundant  I/O 

Avg 

8 

8.3 

8 

8.3 

12.6 

Min 

4 

5 

8 

6 

12 

Max 

13 

13 

8 

9 

13 

Table  3:  Summary  of  page  accesses  for  different  algorithms 


FP  prefers  to  swap-in  the  nodes  with  the  highest  residential  degree  and  swap  out  the  nodes  with  the 
lowest  non-residential  degree.  In  this  example,  these  policies  tend  to  favor  high  degree  nodes  for  buffering, 
leading  to  redundant  I/O  on  low-to-medium  degree  nodes  which  have  come  memory  already.  Note  that 
nodes  R5  and  R.12  have  no  redundant  I/O  despite  a  very  high  degree,  while  R4,  R.2,  and  SO  have 


redundant  I/O  despite  having  a  very  low  degree.  Table  9(Appendix  B.4)  provides  a  detailed  execution 
trace  of  FP  for  interested  readers. 

OM  has  a  good  overall  performance  with  only  5  redundant  I/Os.  However,  the  redundant  I/Os  occur 
with  nodes  of  medium  degree,  e.g.  R4,R6,R7,R8,  and  S7(degree  8.)  The  nodes  with  a  higher  degree 
have  no  redundant  I/O.  This  can  be  related  to  the  swap-in  and  swap-out  policy.  Swap-in  policy  favors 
low-residential  degree  neighbors  of  low  non-residential  degree  in-memory  nodes.  The  policy  has  the 
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unintended  effect  of  pulling  all  of  the  neighbors  (S0.S1.S2.S8)  in  order  of  degree  of  the  in-memory  node 
with  the  lowest  non-residential  degree,  e.g.  R0.  If  one  of  these  neighbors  (S8)  has  a  very  high  degree, 
it  may  stick  around  for  a  long  time  due  to  swap-out  policy  based  on  the  lowest  non-residential  degree*. 
This  favored  treatment  of  high  non-residential  degree  nodes  in  memory  leads  to  increased  redundant 
I/O,  in  this  example.  Table  8(Appendix  B.3)  provides  a  detailed  execution  trace  of  OM  for  interested 
readers. 

Chan’s  method  is  a  logical  refinement  of  the  OM  method.  It  tries  to  bring  in  all  of  the  neighbors 
of  the  node  N  with  the  lowest  non-residential  degree.  This  allows  the  complete  processing  of  all  edges 
incident  on  N.  The  hope  is  to  reduce  redundant  I/O;  however,  Chan’s  method  leads  to  an  unintended 
side  effect  for  spatial  joins.  After  R,0  comes  into  the  buffer,  its  neighbors(S0,Sl,S2,S8)  are  brought  in 
to  process  it.  After  R,0  is  processed,  the  next  node  with  the  lowest  non-residential  degree  is  R,l,  which 
comes  in  memory  with  its  remaining  neighbor  S3.  After  R,1  is  processed,  R,2  is  selected,  and  soon  the 
nodes  of  R,  are  favored  to  be  processed  first,  since  the  first  node  to  be  loaded  was  from  R,.  This  leads  to 
excessive  redundant  loading  for  nodes  of  S,  in  this  example.  Table  7( Appendix  B.2)  provides  a  detailed 
execution  trace  of  Chan  for  interested  readers. 

In  contrast  to  FP,  OM,  and  Chan,  SC  uses  a  global  clustering  approach.  SC  uses  symmetric  clustering 
to  permute  the  rows  and  columns  of  the  adjacency  matrix  representation  of  PCG  to  bring  as  many  edges 
as  possible  within  the  B-diagonal.  All  of  the  edges  within  the  B-diagonal  can  be  processed  without 
redundant  I/O.  The  redundant  I/O  for  edges  outside  the  B-diagonal  are  minimized  by  computing  the 
vertex  cover,  i.e.  a  set  of  nodes  covering  all  of  the  edges  outside  the  B-diagonal.  The  nodes  in  the  vertex 
cover  are  scheduled  with  appropriate  partitions,  and  some  of  these  vertices  may  yield  multiple  redundant 
I/Os.  SC  has  only  three  redundant  I/Os,  resulting  from  a  vertex  cover  {S8,R,12,R,5}  of  the  15  off  B- 
diagonal  edges.  Incidentally,  the  nodes  with  redundant  I/Os  have  a  very  high  degree.  Table  10(  Appendix 
B.5)  provides  a  detailed  execution  trace  for  interested  readers. 

4  Proposed  Symmetric  Clustering  Method 

In  contrast  to  incremental  approaches  (e.g.  FP.  OM.  Chan),  Symmetric  Clustering(SC)  uses  a  global 
approach  based  on  Band-diagonalization  of  the  adjacency  matrix  representation  of  PCG.  The  number 
of  redundant  I/Os  depends  only  on  the  edges  outside  of  the  B-diagonal  and  can  often  be  reduced  via 
identifying  a  vertex-cover. 

Recall  that  pure  B-diagonal  entries  for  a  square  matrix  are  defined  by  {Mpcc[h  j]  =  1}  =>  {\'i  ~~  j\  < 
[B/2\ },  where  B  is  the  number  of  buffers  available  for  pages  of  R.  and  S.  Band-diagonalization  of  a  matrix 
rearranges  the  rows  and  columns  of  the  matrix  to  bring  in  as  many  non-zero  entries  as  possible  within 
B-diagonals.  Thus,  a  matrix  with  only  B-diagonal  elements  is  already  band-diagonalized.  However,  a 
band-diagonalized  matrix  may  have  a  few  entries  outside  of  the  B-diagonal. 


SC  Algorithm 

Input:  G  =  (V, E)  is  a  page-connectivity  graph;  B  is  the  number  of  buffers. 

*It  is  noil-intuitive.  See  the  explanation  from  [28]  “If  a  node  in  the  buffer  is  going  to  be  swapped  out,  then  in  the  worst 
case  it  could  be  brought  in  one  additional  time  for  each  page  it  is  connected  to  outside  t  he  buffer,  assuming  it  were  to  be 
swapped  out  each  time.”  Thus  swap-out  kicks  out  the  node  with  the  lowest  non-residential  degree. 
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Output:  S  =<  Pj  ,  P2,—,Pr  >  is  a  page-access  sequence  with  r  >  |Vj.(P;s  need  not  be  distinct) 

{Step  1}  <  G b  D .  Porder  >=Eand-diagonaiiation(  G.  B);  /*  Get  B -diagonal  graph  and  ordered  set  of  partitions*/ 

{Step  2}  <  Gqbd  >=Get-off-B-diagonai-entries(<JB d )i  /*  Bind  all  the  off- B- diagonal  edges  and  nodes  from  Gbd  */ 
{Step  3}  <  VC  >=Find~vertex~cover(G’o bo):  /*  Find  the  vertex  cover  VC  for  the  Off-B-diagonal  cut-edge  Eqbe  */ 
{Step  4}  <S  >=Access-sequence-generator(Por^lfi. .Gbd-. VC);  /*  Generate  the  page  access  sequence  */ 

First.  SC  derives  the  Band-diagonalized  matrix  Gbd  bv  permuting  the  rows  and  columns  of  the 
adjacency  matrix  representation  of  PCG  to  bring  in  as  many  edges  as  possible  within  the  B-diagonal. 
Secondly,  from  Gbd ,  SC  gets  the  graph  Gobd  for  off-B-diagonal  edges  and  their  corresponding  nodes. 
Thirdly.  SC  determines  the  vertex  cover  for  Gqbd-  Finally,  SC  generates  the  page-loading  sequence 
based  on  the  Band-diagonalized  matrix,  the  vertex  cover  of  the  off-B-diagonal  edges,  and  the  partition 
ordering. 


Example  Revisited 

Consider  the  join- computation  problem  discussed  in  Figures  10  and  11.  The  input  to  the  SC  algorithm 
is  the  page-connectivity  graph  shown  in  Figure  10(b). 

In  step  1,  the  nodes  in  the  PCG  of  Figure  10(b)  are  rearranged  to  get  as  many  edges  with  the 
B-diagonal  as  possible.  The  lightly  shaded  area  in  Figure  10(b)  shows  the  B-diagonal.  In  this  example, 
input  graph  G  and  output  graph  Gbd  of  step  1  are  identical  for  simplicity.  The  nodes  are  partitioned  in 
groups  of  7  nodes,  which  is  1  less  than  the  number  of  memory  buffers  available  for  R,  and  S.  Partitions 
P 1 = { R0-R4 ,S0-S  1 } ,  P2={R,5-R6,S2-S6},P3={R7-R11,S7-S8},  P4={R12-R.13,S9-S13}  are  used  in  this 
example,  as  shown  in  the  shaded  rectangles  in  Figure  10(b).  The  partitions  are  loaded  in  the  order 
P1,P2,P3,  and  then  P4.  The  edges  between  the  nodes  within  a  common  partition  can  be  processed  with 
no  redundant  I/O.  The  edges  between  the  nodes  that  are  in  adjacent  partitions  in  the  loading  sequence 
and  which  fall  inside  the  B-diagonal  can  also  be  processed  without  any  redundant  I/Os,  due  to  the 
availability  of  1  extra  buffer. 

The  redundant  I/Os  for  the  remaining  edges  can  be  reduced  by  computing  the  vertex  cover  via  step 
2  and  3.  There  are  15  edges  off  the  B-diagonal  with  a  vertex  cover  of  {S8,  R5,  R12}.  There  are  5  edges 
between  partitions  PI  and  P3,  and  they  all  incident  on  node  S8.  They  can  be  processed  with  one  extra 
I/O  by  bringing  S8  into  the  last  buffer  when  the  nodes  of  PI  are  in  the  buffer.  Similarly,  the  10  edges 
between  the  nodes  in  P2  and  P4  can  be  processed  in  two  I/Os.  Since  5  of  those  are  incident  on  R,12  and 
R5,  bringing  R12  with  P2  and  R5  with  P4  will  take  care  of  all  of  these  edges.  Thus,  SC  results  in  only 
three  redundant  I/Os,  resulting  from  a  vertex  cover  {S8,R,12,R5}  for  the  15  off  B-diagonal  edges.  Step 
4  is  to  generate  a  page  access  sequence  using  the  execution  trace  from  previous  steps. 

4.1  The  I/O  cost  of  SC  method 

The  symmetric  clustering  approach  to  minimize  redundant  I/O  can  be  described  in  terms  of  the  following 
problem  statement: 

Lemma  2  Given  a  loading  ordered  partition  {  Pj  .  P^ , . . . .  P; .  Pj  , . . . .  P/- }  of  PCG  =  (V,E),  there  is  a 
page  access  sequence  of  length  k  —  |F|+  redundant-I/O  to  compute  the  spatial  join  where  the  redundant¬ 


ly 


I/O  is  given  by: 


E 


Partition  —  degree(Vj ) 


Vj  €(vertx  cover  of  outside  B— diagonal  edges) 


(1) 


where  Partition  —  degree(Vj )  is  the  number  of  distinct  partitions  which  contain  nodes  Vj  outside  of  the 
B-diagonals. 

Proof:  All  of  the  edges  within  the  main  B-diagonals  can  be  processed  without  redundant  I/O,  using  a 
contour  diagonalization  strategy  as  illustrated  in  Figure  9(c).  The  redundant  I/O  for  each  node  in  the 
vertex  cover  is  limited  by  the  number  of  partitions  sharing  an  off- B- diagonal  edge. 


4.2  A  heuristic  for  Band- diagonalization 

The  first  step  of  Band-diagonalization  can  be  based  on  either  specialized  envelope-reduction  algo¬ 
rithms  [1,  9,  23]  or  min-cut  graph  partition  algorithms  [20,  21].  We  use  the  latter  in  this  paper  and  plan 
to  explore  the  former  in  future  work.  We  describe  the  heuristic  approach  that  we  currently  use. 


Band-diagonalization 

Input:  G  is  a  page  connectivity  graph;  B  is  the  number  of  buffers. 

Output:  G no  is  the  B-diagonal  connectivity  graph;  Border  ‘s  the  partition  order 

PSetm  =  Graph-Partition(G.  B  -  1);  /*  Using  graph  partition  software  */ 

Border  =  Order-Partftion(PSe<m);  /*  Order  the  partitions  using  the  greedy  heuristic  */ 

Graph- Partition:  A  min-cut  partition  algorithm,  e.g.  metis  [20],  divides  the  nodes  of  the 
PCG  into  disjoint  subsets  while  minimizing  the  number  of  edges  whose  nodes  are  incident  in  two 
different  partitions.  One  memory  buffer  is  reserved  for  bringing  in  pages  from  the  vertex  cover  of 
the  off-B-diagonal  entries.  For  example,  the  min-cut  partitioning  of  PCG  for  the  overlay(R,S)  in 
Figure  10  may  yield  4  partitions  for  B=8.  The  partitions  shown  in  Figure  10(b)  are  P1={R,O-R,4,S0-S1}, 
P2={R,5-R6,S2-S6},P3={R,7-R,11,S7-S8},  P4={R12-R13,S9-S13},  resulting  in  69  edges  whose  incident 
nodes  are  in  two  different  partitions.  The  breakup  of  these  edges  by  pairs  of  partitions  of  incident  nodes 
is  shown  in  Table  4.  Formally,  the  min-cut  graph-partitioning  algorithm  addresses  the  following  problem: 

Given:  A  connectivity  graph  G  =  (V,  E)  with  |Vj  =  n,  and  the  number  of  buffers,  B  >  2. 

Find:  A  partition  of  V  into  p  subsets,  V\,  V-2,  ■  ■  ■ ,  Vp  such  that  V)  fj  Vj  =  0  for  i  /  j  and  (J;-  V)  =  V. 

Objective:  Minimize  the  size  of  the  set  of  edges  Ec  C  E  whose  incident  vertices  belong  to  different 
subsets. 

Constraint:  |V)|  <  (B  —  1),  and  the  number  of  partitions,  p  =  |jVj/(B  -  1)]. 


Recent  advances  have  provided  scalable  graph-partitioning  software  such  as  Metis  [20],  which  can 
handle  the  large  graphs  relevant  to  databases  in  a  relatively  reasonable  response  time,  e.g.  a  few  seconds. 
We  have  had  good  results  using  it  for  database  problems  [24,  32]. 
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Order-Partition  chooses  a  partition  ordering,  i.e.  a  loading  sequence,  using  a  partition-interaction 
matrix  M.  An  entry  M[Pj,  Pj]  lists  the  number  of  cut-edges  between  the  nodes  in  partitions  P,  and 
Pj.  An  example  partition-interaction  matrix  for  the  join-index  of  Figure  10  is  shown  in  Table  4.  The 
procedure  uses  a  simple  heuristic  to  construct  the  loading  sequence.  It  sorts  the  entries  in  M[Pj,Pj] 
in  descending  order  and  arbitrarily  breaks  the  tie.  It  initially  chooses  the  entry  with  the  largest  value, 
getting  a  loading  sequence  of  length  2.  Then,  it  extends  the  loading  on  both  sides  in  a  greedy  manner. 
For  example,  suppose  M[P$ ,  P2]  was  selected  first.  Then,  the  loading  sequence  Pi-P\  can  be  extended  to 
the  right  by  choosing  Pj  and  extended  to  the  left  by  choosing  Pj  from  among  the  remaining  partitions. 
The  choice  is  based  on  the  highest  value  of  M[P2.Pj]  and  M[P$.Pj]  in  the  partition-interaction  matrix. 
A  better  heuristic  can  be  designed  to  select  loading  sequences  that  have  a  higher  number  of  cut-edges 
between  consecutive  partitions.  To  improve  the  performance  of  the  proposed  SC  method,  we  will  consider 
these  in  future  work. 


Pi 

P2 

P3 

Pa 

Pi 

* 

18 

5 

0 

P'l 

18 

i= 

18 

10 

p* 

5 

18 

* 

18 

Pt 

0 

10 

18 

* 

Table  4:  #  of  cut-edges  between  partitions  Pj  and  Pj 


We  use  Figure  12  as  an  example  to  illustrate  the  steps  in  B and- di agonaliz  ation  using  the  graph- 
partition  technique.  Figure  12(a)  is  the  original  PCG  relation,  where  R  and  S  are  two  relations  to 
be  joined,  and  where  each  point  in  the  picture  denotes  an  edge  connection  between  the  pages  of  two 
relations.  We  use  Metis  [20]  to  partition  this  PCG,  and  each  partition  has  size  (B  -  1),  where  B  is  the 
number  of  buffers  available.  We  show  the  result  after  the  partition  in  Figure  12(b):  the  pages  from  the 
R,  and  S  relations  are  relabeled  by  their  partitions.  Finally,  we  re-order  these  partitions  to  bring  as  many 
points  as  possible  inside  the  B-diagonal,  as  shown  in  Figure  12(c).  In  Figure  12(b),  28%  of  the  edges 
are  outside  the  B-diagonal.  After  partition  reordering,  we  have  reduced  these  edges  to  22%  of  the  total 
edges.  Edges  outside  of  the  B-diagonal  can  be  processed  by  using  a  vertex  cover,  as  discussed  next. 

4.3  Vertex  Cover  Computation 

The  redundant  I/Os  in  the  SC  approach  are  due  to  the  edges(i.e.  non-zero  matrix  elements)  outside 
the  B-diagonal  of  a  clustered  adjacency-matrix  representation  of  a  join-index.  These  outside-B-diagonal 
edges  are  grouped  via  a  vertex-cover  algorithm  to  determine  a  small  set  of  pages  needing  redundant 
I/Os.  Determining  the  minimal  vertex-cover  for  a  general  graph  is  NP-hard  [8].  However,  polynomial 
time  algorithms  [14]  are  available  for  determining  minimal  vertex  covers  for  bi-partite  graphs,  e.g.  PCG 
for  join -in  dices. 

The  Find-vertex-cover  procedure  determines  the  vertex  cover  for  all  of  the  off-B-diagonal  edges  by 
a  first  but  greedy  heuristic  which  is  described  below.  The  heuristic  sorts  the  nodes  related  to  the  off- 
B-diagonal  edges  by  their  degree,  i.e.  the  number  of  incident  edges.  The  node  with  the  highest  degree 
is  added  to  the  vertex  cover  and  all  of  the  edges  incident  on  this  node  are  dropped.  These  steps  are 
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(a)  The  original  PCG  adjacency 
matrix,  from  a  spatial  join-index  for 
the  Sequoia  2000  dataset 


(b)  After  graph  partitioning,  28% 
of  the  edges  are  outside  the  B- 
di agonal  ( B=  1 00) 


(c)  After  ordering  partitions.  22% 
of  the  edges  are  outside  the  B- 
di agonal  (B— 100) 


Figure  12:  Using  graph  partitioning  to  derive  the  B-diagonal 


repeated  to  cover  all  of  the  off-B-diagonal  edges.  In  the  future,  we  plan  to  use  better  algorithms  which 
are  likely  to  improve  the  performance  of  the  proposed  SC  method. 


Find- vertex-cover 

Input:  Go  fin  =  (VobD)Eobd)  is  the  graph  for  off-B-diagonal  edges  and  corresponding  nodes 
Output:  VC  is  the  vertex  cover  for  Gqbd 

whilefEtfBD  is  not  empty)  { 

'/ftiu/i«f=Fincl-highest-degree-node(Vonr));  /*  Node  Vhighest  hat  the  highest  degree  */ 

VC  =  VC\J  Vhiahest;  /*  Add  this  node  to  the  set  of  vertex  cover  */ 

Update(C?oBD);  /*  Update  Gqbd  by  removing  node  Vhiijhcat  its  corresponding  edges  */ 

} 

In  Figure  10.  for  example,  with  the  loading  sequence  Pi  —  P-2  —  Pi  —  Pa  ,  the  vertex  cover  for  the  15 
off-B-diagonal  cut-edges  are  nodes  R5,  R.12  and  S8.  Page  R5  covers  edges  (R5,S9),  (R5.S10),  (R,5,S11), 
(R,5,S12)  and  (R,5,  S13).  Page  R.12  covers  edges  (R.12.S2),  (R.12, S3),  (R,12,S4),  (R12,S5)  and  (R,12,S6). 
Page  S8  covers  edges  (R.0,S8),  (R.1,S8),  (R.2.S8),  (R.3,S8)  and  (R.4,S8). 

4.4  Generate  access  sequence 

The  Access-sequence-generator  procedure  derives  the  page- access  sequence.  It  loads  each  partition  in 
the  pre-detcrmined  order.  When  transferring  from  one  partition  Pj  to  the  next  scheduled  partition  F,:+i, 
the  procedure  orders  the  loading  sequence  of  the  nodes  using  the  contour-diagonalization  order  shown  in 
Figure  9(c).  After  loading  a  whole  partition,  we  find  all  of  the  off-B-diagonal  vertex  cover  nodes  which 
connect  to  this  partition,  and  load  these  nodes,  one  by  one,  to  compute  the  join. 


19 


Access-sequence-generator 

Input:  Porder  is  the  loading  sequence  of  the  partitions; 

Gbd  is  the  B-diagonai  connectivity  graph; 

VC  is  the  vertex  cover  for  all  the  Off-B-diagonal  edges. 

Output:  S  =<  Pi  ,  /J2. ....  Pr  >  is  a  page  access  sequence. 

for(f  =  1 ;  i  <  I  Porderl)  *  +  +){ 

P:=  GetPartition(Por<jcs-,f)  /*  Get  the  ith  partition  */ 

if(i==l)  {  AddPageSequencef.S'.  Pi);/*  Add  all  the  nodes  within  Pi  into  the  loading  sequence  */  } 
else  {  OrderAndAddPageSequence(5. 

/*  Order  and  add  the  nodes  within  Pi  into  the  loading  sequence  hy  the  following  rules:  */ 

/*  i.  Add  the  node  within  Pi  which  has  the  highest  connectivity  with  Pi-iinB  —  diagonal  */ 
/*  2.  Replace  the  node  within  P;_i  which  has  finished  its  join  with  the  nodes  in  Pi  */ 

} 

PVC-Set  =  FindConnected.node_from_VC(P»,VrC); 

/*  Find  if  any  Off-B-diagonal  vertex  cover  which  connects  to  this  partition  */ 
AddPageSequence(S,  PVC-Set);  /*  Add  these  nodes  into  the  loading  sequence  */ 


5  Comparative  Evaluation  of  SC,  AC  and  Competitors 


The  experimental  setup  is  shown  in  Figure  6.  Recalling  that  the  dataset  is  extracted  from  Sequoia 
2000  [33],  we  construct  join-indices  for  N-nearest-neighbor(Q.A),  as  well  as  for  distance- based  range 
queries(Q.B).  Variable  parameters  include  buffer  size,  page  size,  and  edge  ratio.  Relation  R.  refers  to 
the  GNIS  Point,  table,  and  relation  S  refers  to  the  Landuse  Polygon  table  from  the  Sequoia  2000  [33] 
dataset.  For  the  sake  of  brevity,  we  refer  readers  to  section  2.3.1  for  details  of  the  experiment  design. 

5.1  Experiment  results 


Figures  13  and  14  compare  all  of  the  OPAS-FB  heuristics  for  N-nearest-neighbor  join-indices  and  range- 
query  join-indices,  respectively.  For  different  experiments,  we  vary  the  page  size,  buffer  size,  and  edge 
ratio. 


5.1.1  Page  size 

Page  size  affects  the  clustering  of  the  base  relations  and  also  the  degree  of  the  nodes  in  the  PCGs.  We 
study  the  effect  of  page  size  on  the  performance  of  the  OPAS-FB  methods.  Figure  13(a)  and  14(a)  show 
the  effect  of  page  size.  The  page  size  is  varied  from  2  kbytes  to  64  kbytes.  The  page  size  and  the  number 
of  page  accesses  are  shown  in  the  logarithmic  form  of  base  two.  In  the  N-nearest-neighbor  join-indices, 
the  SC  and  AC  outperform  all  of  the  other  methods  using  different  page  sizes.  In  the  Range-query 
join-indices,  SC  and  AC  require  fewer  page  accesses  than  all  of  the  other  methods.  The  AC  outperforms 
SC  when  page  size  is  greater  than  64  kbytes.  The  OMJs  method  performs  well  with  the  4  and  8  kbyte 
page  size. 


20 


log(Page  size  (Kbytes))  (Buffer  size/smaller  (size  of  R,  size  of  S))*100  Number  of  nearest  neighbors 

(a)  Page  size:  2K-64K,  Buffer  size:  (b)  Buffer  size:  5%  -  20%  of  the  (c)  No  of  nearest  neighbors=  .l- 

5%  of  the  smaller  relation,  No  of  smaller  relation.  Page  size:  2K,  No  5(0=0.002-0.005),  Buffer  size:  5% 

nearest  neighbors=3  of  nearest  neighbors=3(0=O.OO3)  of  the  smaller  relation,  Page  size: 

2K 

Figure  13:  Nearest  Neighbor  Query:  The  effect  of  page  size,  buffer  size,  and  number  of  nearest  neighbors 
for  different  OPAS-FB  heuristics 

5.1.2  Buffer  size 

Figure  13(b)  and  14(b)  show  the  effect  of  buffer  size.  We  varied  the  number  of  buffers  as  a  percentage  of 
the  number  of  pages  of  the  smaller  relation.  The  percentage  is  changed  from  5  to  20.  The  AC  and  SC 
perform  better  than  all  of  the  other  methods  when  the  buffer  size  is  relatively  low,  e.g.  5  to  10  percent. 
The  Chan  and  OM  methods  do  well  with  large  buffer  sizes. 

5.1.3  Edge  Ratio 

In  this  experiment,  we  changed  the  edge  ratio  by  increasing/decreasing  the  number  of  neighbors  and  the 
size  of  the  range  query.  The  result  of  the  experiment  is  shown  in  Figures  13(c)  and  14(c).  AC  and  SC 
uniformly  outperform  all  of  the  other  methods.  The  SC  requires  fewer  page  accesses  than  AC. 

6  Conclusion  and  Future  Work 

In  this  paper,  we  introduced  two  new  algorithms  for  spatial  join  computation,  given  a  join-index  and 
a  fixed  buffer  size.  The  key  idea  is  to  use  spatial  clustering.  The  propsed  AC  and  SC  algorithms 
outperformed  the  traditional  methods  in  our  experiments  with  the  Sequoia  2000  [33]  dataset,  particularly 
when  the  size  of  the  memory  buffer  was  small (<  10%),  relative  to  the  size  of  the  spatial  relations.  We 
also  provided  a  formal  characterization  of  an  upper  bound  on  the  number  of  redundant  I/Os  needed  by 
AC  and  SC. 

In  the  future  we  would  like  to  improve  some  of  the  heuristics  chosen  in  the  implementation  of  AC 
and  SC,  as  discussed  throughout  this  paper.  We  would  also  like  to  look  at  related  issues  regarding 
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log(Page  size  (Kbytes)) 

(a)  Page  size:  2K  -  64K,  Buffer  size: 
10%  of  the  smaller  relation,  Range 
query  size  =  3800 


(b)  Buffer  size:  5%  -  20%  of  the 
smaller  relation,  Page  size:  2K, 
Range  query  size  =  3800(0=0.003) 


Size  of  range  query 

(c)  Range  query  size  =  400  - 
4800(0=0.002-0.003),  Buffer  size: 
10%  of  the  smaller  relation,  Page 
size:  2K 


Figure  14:  Range  Query:  Effect  of  page  size,  buffer  size,  and  number  of  nearest  neighbors  for  different 
OPAS-FB  heuristics 


maintenance  of  join-indices  in  the  face  of  updates  and  also  the  interaction  of  join-indices  with  the  join- 
computation  algorithm.  Finally,  we  are  interested  in  exploring  the  usefulness  of  AC  and  SC  in  data 
Warehouses  [15,  16]  (e.g.  processing  star-joins  using  the  STARindex  [15])  and  spatial  data  mining(e.g. 
neighborhood  index  [6]). 
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A  Summary  trace  for  different  algorithms  in  Figure  11 

Table  5  provides  a  summary  trace  of  various  heuristics.  Each  row  represents  a  new  page  being  fetched 
into  main  memory.  Thus,  the  number  of  rows  represent  the  total  number  of  pages  fetched.  For  simplicity, 
the  clustering  of  I/O  for  multiple  pages  is  not  modeled.  An  entry  (+R0)  for  Sorting  in  iteration  1  means 
that  page  RO  was  fetched.  An  entry  (+51,  =  51)  for  iteration  9  of  Sorting  implies  that  51  was  fetched 
into  memory (+51)  and  that  all  edges  incident  on  51  were  processed  with  the  pages  available  in  the 
buffer.  The  set  of  pages  available  in  the  buffer  in  this  iteration  are  {.RO,  Rl,  R2, . . . ,  R6},  since  we  have 
fetched  those  in  previous  iterations.  The  buffer  containing  51  can  be  reused  in  the  next  iteration  to  bring 
in  52,  as  shown  by  entry  (+52)  for  Sorting  in  iteration  10.  The  next  interesting  entry  is  (—52.  +53)  in 
iteration  11  for  Sorting,  where  the  buffer  containing  52  is  overwritten  by  incoming  page  53  even  though 
some  edges  (e.g.  R7  —  52)  incident  cannot  be  processed  right  away.  This  is  due  to  a  buffer  size  fixed  at 
eight.  Note  that  52  returns  to  memory  in  iteration  29  to  process  the  edge  (R 7  —  52).  This  leads  to  a 
redundant  I/O.  Because  the  graph  is  highly  connected  in  spatial  order,  the  AC  will  not  re-cluster  the  R 
relation,  and  uses  the  same  loading  sequence  {is  the  Sorting  heuristic. 

Traces  of  other  algorithms  are  shown  in  other  columns  using  8  buffers.  Note  that  the  number  of  the 
last  rows  with  a  '+'  entry(page  fetch)  designates  the  total  number  of  page  I/Os  for  an  algorithm.  In 
other  words,  the  Sorting/AC-based  algorithm  has  40  I/Os,  Chan's  heuristic  has  35,  OM  has  33,  FF  has 
45  and  SGP  has  31,  as  shown  in  the  last  row  labeled  I/O  count. 
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-R7,+R  1 1 ,— S8 

22 

-S13.+R7,— R1 

+R10 

+S10,=S10 

23 

+R8.-R2 

-S5.+S1 2,— R 1 0 

+S9 

24 

+R9,=R3 

+R11 

-R8,+R7,=S9 

25 

+R10,=R4 

-S0.4-S13,— Rll 

+S7 

20 

+R11,=R5 

+R13.-R13 

-S7,-f  SO,— R7 

27 

OFFIB 

+R 1 2,— S8 

+S1 1 ,— R9,— SI  1 

28 

+R13 

+  R5,-S7 

KEffiJSiSIDSaH 

29 

+S2.-S2 

+S0,-S0,=S9 

+S13,=R5,=R11 

30 

+S3.-S3 

+S1,— Si ,— S10 

-f  R4,— R4,— R 1 2 

31 

+S4,=iS4 

+S2,=S2,=S11 

32 

+S5.-S5 

+S3,=S3,=S12 

+R8,=S0 

33 

+S0.-S0 

+S4,=S4,=S13 

+S7 

34 

+S7,=S7 

+S5,— S5 

35 

+S8.-S8 

+S0 

30 

+S9.-S9 

37 

+S10.-S10 

38 

+S11,— Sll 

39 

+S1 2,— S12 

40 

+S13 

41 

42 

43 

44 

45 

mmu.mm 

40 

35 

33 

Table  5:  Summary  of  results  for  all  algorithins(Buffei 


B  Trace  for  different  algorithms  in  Figure  11 

B.l  Execution  Trace  for  ASC/Sorting  method 

Table  6  shows  the  behavior  of  the  Sorting  method  for  computing  a  join,  given  the  join-index  of  Figure  10. 
Table  6  has  five  columns.  The  first  column  shows  the  iteration  number.  The  second  column  shows  the 
node  swapped  out  in  the  current  iteration.  The  third  column  shows  the  node  selected  and  brought  into 
the  memory  buffers.  The  fourth  and  fifth  columns  show  the  pages  of  R  and  S  in  the  main  memory 
buffer.  The  last  column  shows  the  nodes  which  have  been  processed  completely  and  need  not  come  into 
the  memory  buffer  again. 
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RO 

RO 
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0 
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SO 

SO 

9 
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SI 

SI 

1.0 

S2 

R  0,R  1  ,R2  ,R  3  ,R4  ,R  5 ,  R  0 

S2 

1 1 

S2 

H-2M 

wmsxmsmz*aa\  \  m  i 

S3 

12 

S3 

S4 

S4 

13 

S4 

R  0,R  1  ,R  2 ,  R  3  ,R  4  ,R  5 ,  R  0 

So 

14 

S5 

— 

R  0,R1  ,R  2  ,R  3  ,R  4  ,R  5 ,  R  0 

so 

15 

SO 

k um 

S7 

16 

S7 

■aaaKMguTwiit’TMWi 

S8 

17 

S8 

R  0,R  1  ,R2 ,  R  3  ,R4  ,R5,R6 

S9 

18 

S9 

KHil 

R  0,R  1  ,R  2 ,  R  3  ,R  4  ,R  5 ,  R  0 

S10 

19 

S10 

Sll 

Sll 

R  0,R  1  ,R  2 ,  R  3  ,R  4  ,R5,R0 

SI  2 

21 

S12 

R  0,R  1  ,R  2 ,  R  3  ,R  4  ,R  5 ,  R  0 

SI  3 

R0,R  1  ,R2,R3,R4,R5,R0 

22 

S13 

msm 

R7 

'23 

24 

■ 

M  a#  tfJ 

25 

■MB 

R7,R8,R9,R10 

26 

MfflM 

R7,R8,R9,R  1 0,R1 1 

27 

■ 

28 

■:TM 

mmmmmammasm 

29 

m 

R7,R8,R9,R10,R  11, R1 2,R  1 3 

S2 

S2 

30 

■KM 

R7,R8,R9,R  10,R  11, R1 2,R  1 3 

S3 

S3 

31 

mzm 

S4 

S4 

32 

mism 

R  7,R8,R9,R10,R1 1  ,R12,R  13 

S5 

S5 

33 

R7  ,R8  ,R9 ,  R 1 0  ,R  1 1  ,R  1 2 ,  R 1 3 

SO 

SO 

34 

S7 

R  7  ,R8  ,R9 ,  R 1 0  ,R  1 1  ,R  1 2 ,  R 1 3 

S7 

S7 

35 

R7,R8,R9,R10,R11,R12,R13 

S8 

S8 

30 

mtsm 

R7,R8,R9,R10,R  11, R1 2,R  1 3 

S9 

S9 

37 

gcITM 

R7,R8,R9,R10,R  11, R1 2,R  1 3 

S10 

S10 

38 

EM 

R  7  ,R  8  ,R9 ,  R 1 0  ,R  1 1  ,R  1 2 ,  R 1 3 

SI  1 

Sll 

39 

EM 

R7,R8,R9,R10,R  11, R1 2,R  1 3 

SI  2 

SI  2 

40 

SI  3 

R  7  ,R8  ,R9 ,  R 1 0  ,R  1 1  ,R  1 2 ,  R 1 3 

SI  3 

R7 ,  R8 ,  R9 ,  R 1 0 ,  R 1 1 ,  R 1 2 ,  R 1 3  ,S  1 3 

Table  6:  Example:  the  Sorting/ ASC  method(Buffer  size =8) 
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B.2  Execution  Trace  for  Chan’s  method 


Table  7  shows  the  behavior  of  Chan’s  method  for  computing  a  join,  given  the  join-index  of  Figure  10. 
Table  7  has  five  columns.  The  first  column  shows  the  iteration  number.  The  second  column  shows  the 
node  swapped  out  in  the  current  iteration.  The  third  column  shows  the  node  selected  and  brought  into 
the  memory  buffers.  The  fourth  and  fifth  columns  show  the  pages  of  R,  and  S  in  the  main  memory 
buffer.  The  last  column  shows  the  nodes  which  have  been  processed  completely  and  need  not  come  into 
the  memory  buffer  again. 
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K  Bolter 

S  Buffer 
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2 

SO 

Ho 
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SI 

RO 
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4 
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RO 

S0.S1.S2 

5 

S8 

RO 
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R0 

6 

mm 

HI 
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7 

Rl 
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Rl 

8 

R2 

S0.S1.S2.S8, S3 

9 

R2 
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R2 
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R3 

SO, SI  ,S2,S8,S3,S4,S5 

R3 

12 

R4 

SO, SI  ,S2,S8,S3,S4,S5 

13 

SO 

R4 

S1,S2,S8,S3,S4,S5,S6 

R4 

14 

mum 

■J32Mi§Ef» 

S1,S2,S8,S3,S4,S5,S6 

15 

SI 

mm 

R(i 

S2,S8,S3,S4,S5,S6,S7 

R0 

16 

mm 

R7 

S2,S8,S3,S4,S5,S6,S7 

17 

S2 

msm 

R7 

S8,S3,S4,S5,S6,S7,S9 

R7 

18 

■iiiM-lll 

S8,S3,S4,S5,S6,S7,S9 

19 

S3 

-aa— 

■Tf  ill 

S8,S4,S5,S6,S7,S9,S10 

H8 

20 

ans 

R9 

S8,S4,S5,S6,S7,S9,S10 

21 

S4 

R9 

S8,S5,S6,S7,S9,S10,S1 1 

R9 

22 

■  SUM 

■  Ml— 1 

S8,S5,S6',S7,S9,S10,S1 1 

23 

S5 

*5051 

riw  i 

S8,S6,S7,S9,S10,S1 1  ,S12 

R10 

24 

Rl  1 

S8,S6,S7,S9,S10,S11,S12 

25 

S(i 

■SIM 

Rll 

S8,S7,S9,S10,S1 1  .S12.S13 

Rll 

26 

mm 

i'TT—T 

S8,S7,S9,S10,S1 1  .S12.S13 

HT3 

27 

M\\  m  i  ii 

S8,S7,S9,S10,S11,S12,S1 3 

S8 

28 

■lid— 

R5.R12 

S7,S9,S10,S1 1  .S12.S13 

S7,S9,S10,S1 1  ,S  12.S13 

29 

R5.R12 

SO 

SO 

30 

—Hi  ill;' 

SI 

SI 

31 

S2 

S2 

32 

R5.R12 

S3 

S3 

33 

S4 

R5.R12 

S4 

S4 

34 

1 

S5 

S5 

35 

SO 

R12,R5,S0 

Table  7:  Example:  Chan’s  method  (Buffer  size =8) 
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B.3  Execution  Trace  for  OM’s  method 


Table  8  shows  the  behavior  of  OM’s  method  for  computing  a  join,  given  the  join-index  of  Figure  10. 
Table  8  has  five  columns.  The  first  column  shows  the  iteration  number.  The  second  column  shows  the 
node  swapped  out  in  the  current  iteration.  The  third  column  shows  the  node  selected  and  brought  into 
the  memory  buffers.  The  fourth  and  fifth  columns  show  the  pages  of  R,  and  S  in  the  main  memory 
buffer.  The  last  column  shows  the  nodes  which  have  been  processed  completely  and  need  not  come  into 
the  memory  buffer  again. 
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Swap  out 
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R  Boiler 

S  Buffer 
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1 
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RO 

2 

SO 

RO 

SO 

3 

SI 

RO 

SO, SI 

4 

S2 

RO 

S0.S1.S2 

5 

S8 

RO 

S0.S1.S2.S8 

R0 

6 

mm 

S0.S1.S2.S8 

7 

R1 

S0.S1  .S2.S8.S3 

R1 

8 

R2 

S0.S1.S2.S8, S3 

9 

tel* 

R2 

S0.S1  ,S2,S8,S3,S4 

R2 

1.0 

S0,Sl,S2,S8,S3,S4 

1 1 

R3 

S0,S1,S2,S8,S3,S4,S5 

R3 

12 

KOI 

R4 

S0,S1,S2,S8,S3,S4,S5 

13 

R4 

R5 

R5 

S0.S1.S2.S8, S3, S4.S5 

SO 

14 

R5,R(i 

S1,S2,S8,S3,S4,S5 

SI 

15 

mm 

■  iimi 

S2,S8,S3,S4,S5 

16 

R6 

■  :TFM 

R5,R7,R12 

S2,S8,S3,S4,S5 

S2 

17 

■  ; M 

R5,R7,R12,R8 

S8.S3.S4.S5 

S3 

18 

BiM 

S8.S4.S5 

S4 

19 

■  3M 

mmstiSHmmsssmm 

S8.S5 

S5 

20 

m 

R5.R7.Rl  2,R8,R9,R  1 0,R  1 3 

S8 

21 

R7 

BSMHB 

R  5  ,R  1 2  ,R  8 ,  R  9  ,R  1 0.R  1 3 ,  R 1 1 

S8 

S8 

22 

R5,R12,R8,R9,R10,R13,R11 

S10 

S10 

23 

— 

R  5  ,R  1 2  ,R8 ,  R9  ,R  1 0,R1 3 ,  R 1 1 

S9 

24 

R8 

mm 

R  5  ,R  1 2  ,R9 ,  R 1 0.R  1 3  ,R  1 1 ,  R7 

S9 

S9 

25 

mm 

R5,R  1 2.R9.R  1 0,R  1 3,R  1 1,R7 

S7 

S7 

mtnm 

R5,R  1 2.R9.R  1 0,R  13,R  1 1  ,R7 

S(i 

R7 

27 

R  5  ,R  1 2,R9 ,  R 1 0,R1 3  ,R  1 1 

Sti.Sl  1 

R9.S1 1 

28 

mmm 

R5,R12,R10,R13,R1 1 

S6.S12 

R10, SI  2 

29 

KM 

R5,R12,R13,Rl  1 

S6.S13 

R5.R  1 1  ,R  12,R  13.S1 3 

30 

mzu 

R4 

S(i 

R4 

31 

SO 

32 

■  M 

R6.R8 

so 

SO 

33 

S7 

R6.R8 

S7 

R0.R8.S7 

Table  8:  Example:  OM’s  methodfBuffer  size  =8) 
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B.4  Execution  Trace  for  FP’s  method 


Table  9  shows  the  behavior  of  FP;s  method  for  computing  a  join,  given  the  join-index  of  Figure  10. 
Table  9  has  five  columns.  The  first  column  shows  the  iteration  number.  The  second  column  shows  the 
node  swapped  out  in  the  current  iteration.  The  third  column  shows  the  node  selected  and  brought  into 
the  memory  buffers.  The  fourth  and  fifth  columns  show  the  pages  of  R,  and  S  in  the  main  memory 
buffer.  The  last  column  shows  the  nodes  which  have  been  processed  completely  and  need  not  come  into 
the  memory  buffer  again. 


Iteration 

Swap  out 

Add 
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R0.R1  ,R2,R3 

S0,S1,S2,S8 

R0 

9 

S0,S1,S2,S8 

10 

El 

■:f i 

R2,R3,R4,R6 
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R8 
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21 
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E9 

■iff 
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S8 
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R5.R13 

86, Si 
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36 
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R5.R10 

S(i 
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38 

R5 
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39 
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R5.R1 

86, S3 
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40 

S4 

R5 

86, S4 

R5 

41 

R2 

R2 

S6,S4 

R2.S4 

42 

R6 

R6 
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S6 
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S7 

R6 

S7 
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44 

R3 

R3 

45 

S5 

S5 
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Table  9:  Example:  FP!s  met  hod  (Buffer  size  =8) 
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B.5  Execution  Trace  for  SC 


Table  10  shows  the  behavior  of  SC's  method  for  computing  a  join,  given  the  join-index  of  Figure  10. 
Table  10  has  five  columns.  The  first  column  shows  the  iteration  number.  The  second  column  shows 
the  node  swapped  out  in  the  current  iteration.  The  third  column  shows  the  node  selected  and  brought 
into  the  memory  buffers.  The  fourth  and  fifth  columns  show  the  pages  of  R.  and  S  in  the  main  memory 
buffer.  The  sixth  column  shows  the  nodes  which  have  been  processed  completely  and  need  not  come 
into  the  memory  buffer  again.  The  last  column  shows  the  partition  number. 
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H9 

4 

27 

mm 

H10,Rll,Hl2 

S7,S8,S9,S10,SU 

S7 

4 

28 

mnvm 
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Table  10:  Example:  SC  methodfBuffer  size  =8) 
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